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Abstract 


This  report.  User  Service  Requirements-Software  Support,  examines  the 
support  delivery  and  performance  requirements  of  users  of  systems  and 
applications  software  from  the  following  vendors:  ADR,  Cullinet,  DEC, 
IBM,  McCormack  &  Dodge,  MSA,  NCA,  Pansophic,  Sterling,  and 
UCCEL.  The  sample  of  over  300  users  is  broken  down  and  analyzed  on 
a  vendor-by- vendor  basis. 

The  report  begins  by  examining  contractual  and  ad  hoc  software  support 
delivery  from  each  vendor.  Each  product  is  analyzed  regarding  its  free- 
dom from  major  and  minor  software  problems  as  well  as  the  vendor's 
ability  to  quickly  resolve  each  problem.  Next,  each  vendor's  ability  to 
measure  and  then  satisfy  its  own  users'  needs  for  specific  software  sup- 
port services,  such  as  telephone  support,  documentation,  and  training,  is 
studied.  Lastly,  each  product's  user  sample's  willingness  to  consider 
alternative  (third-party)  support  sources  is  measured. 

In  addition,  an  analysis  of  the  entire  sample's  attitudes  about  and  satisfac- 
tion with  the  software  support  received  is  presented  as  a  source  of  com- 
parison, not  only  for  each  individual  vendor  performance  analysis,  but 
also  for  last  year's  software  support  user  analysis  as  well. 

This  report  consists  of  1 12  pages,  including  83  exhibits. 
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Introduction 


A  

Scope  Th^  importance  of  software  support  in  the  overall  data  processing  opera- 

tions of  users  is  growing  at  an  extremely  fast  rate  as  the  hardware  compo- 
nents of  systems  become  more  reliable  and  as  applications  performed 
become  more  sophisticated.  Software  problems  constituted  30%  of  aU 
large  system  interruptions  and  20%  of  all  small  system  interruptions  in 
input's  1987  User  Service  Requirement  research,  and  INPUT  expects 
that  these  percentages  will  grow  in  1988. 

This  report.  User  Service  Requirements -Software  Support,  is  the  first 
deliverable  in  a  series  of  reports  that  INPUT  is  preparing  on  software 
support.  The  report  analyzes  software  support  needs  of  users  of  ten 
vendors'  software  packages.  Rather  than  simply  reporting  user  satisfac- 
tion with  the  support  that  they  receive  from  their  vendor,  this  report 
attempts  to  define  what  levels  of  support  each  vendors'  own  user  sample 
requires  as  well  as  how  effectively  each  vendor  is  meeting  those  require- 
ments. In  this  way,  the  report  highlights  areas  of  opportunity  for  each 
vendor,  in  terms  of  both  support  improvement  and  new  support  revenue 
sources. 

B  

Methodology  The  basis  for  this  report  was  the  analysis  of  telephone  interviews  with 

302  software  users,  broken  down  by  product  used  in  Exhibit  I-l.  This 
sample  represented  an  increase  in  size  of  almost  40%  over  1986.  Exhibit 
n-2  breaks  down  the  user  sample  by  industry  served. 

Each  user  was  interviewed  using  the  questionnaire  located  in  Appendix  A 
at  the  end  of  this  report.  Respondent  tides  typically  were  data  processing 
manager,  operations  manager,  or  software  (applications)  manager.  Each 
interview  lasted  approximately  twenty  minutes. 


FUSW 


e  1987  by  INPUT.  Reproduction  Prohibited. 


1 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT 


INPUT 


SAMPLE  BY  VENDOR 


 , 

SOFTWARE 
VENDOR 

INTERVIEWS 

ADR 

30 

CuIIinet 

30 

DEC 

31 

IBM 

30 

McCormack  &  Dodge 

30 

MSA 

30 

NCA 

30 

Pansophic 

30 

Sterling 

31 

UCCEL 

30 

Total 

302 
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SAMPLE  BREAKDOWN  BY  INDUSTRY  SERVED 


INDl  I9TRY 
SERVED 

MI  IMRFR  OF 

RESPONDENTS 

Manufacturing 

128 

Banking  &  Finance 

28 

Services 

24 

Distribution 

23 

Insurance 

20 

Governnnent 

19 

Medical 

17 

Utilities 

13 

Education 

12 

Transportation 

7 

Telecommunications 

6 

Others 

5 

Total 

302 

All  interviews  were  then  entered  into  a  dBASE  III  data  base  and  prepared 
for  statistical  analysis  using  ABSTAT  (a  popular  statistical  package). 

A  list  of  definitions  of  terms  used  in  this  report  is  located  in  Appendix  B  at 
the  end  of  the  report. 
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II 


Executive  Summary 


This  Executive  Summary  is  provided  as  a  convenient  overview  of  the 
main  points  of  this  report.  It  is  presented  in  a  format  that  facilitates  the 
use  of  this  section  as  a  short  presentation,  with  each  exhibit  accompanied 
by  text  that  acts  as  a  script  detailing  the  key  findings  of  the  study. 

This  is  the  second  year  that  INPUT  has  analyzed  the  support  require- 
ments and  experiences  of  users  of  applications  and  systems  software 
(other  than  the  operating  system,  which  is  covered  in  the  User  Service 
Requirements  reports  for  large  and  small  systems).  Last  year,  the  sample 
indicated  that  support  performance  received  by  the  user  sample  met  the 
needs  of  the  users  in  all  areas  except  for  documentation. 

In  1987,  users  also  reported  dissatisfaction  with  the  documentation. 
However,  user  satisfaction  with  training  and  consulting  dropped  consid- 
erably in  1987,  suggesting  that  user  requirements  for  these  services  are 
increasing  at  a  faster  rate  than  vendors  are  able  to  provide. 

On  the  other  hand,  user  satisfaction  with  the  phone  support  that  they 
received  increased,  indicating  a  realization  of  the  cost  effectiveness  of 
this  delivery  mode  of  software  support 


Software  support  delivery  differs  from  hardware  maintenance  in  that 
software  users  expect  (and  receive)  their  support  predominantiy  via  the 
telephone  rather  than  on-site.  Exhibit  11- 1  shows  that  66%  of  the  soft- 
ware support  sample  received  phone  support  contractually,  versus  31% 
who  contracted  for  on-site  support  This  reliance  on  phone  support, 
typically  in  the  form  of  "hodine"  service,  provides  the  user  with  fast 
response  to  questions  or  problems  without  the  additional  burden  of  on- 


Software  Support 

Delivery 

Methodology 
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EXHIBIT  11-1 


SOFTWARE  SUPPORT  DELIVERY 


SUPPORT 
DELIVERY 


USERS  (Percent) 


On-Site  Support 


Phone  Support 


Training 


Access  to 
Problems 
Data  Base 


0 


20 


40 


60 


80 


100% 


Do  Not 


Contractually     ^  Ad  Hoc  Basis  Receive 


site  support  costs.  The  vendor  benefits  from  not  having  to  incur  the 
steady  costs  of  sending  software  engineers  to  support  a  dispersed  product 
base. 


Interestingly,  almost  nine  out  of  ten  of  the  sample  had  some  experience 
with  on-site  support  (on  an  ad  hoc  basis).  Most  likely,  these  users  paid 
hourly  (programming)  fees  to  receive  this  support,  although  in  some  cases 
these  on-site  visits  may  have  been  scheduled  "good  will"  visits  performed 
at  no  charge. 

Phone  support  can  be  an  extremely  effective  support  offering  if  offered  in 
conjunction  with  access  to  a  problems  data  base.  However,  less  than  half 
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of  the  software  users  surveyed  contractually  received  this  support  service. 
On  a  positive  note,  this  usage  is  up  significantly  from  last  year's  re- 
sponse, which  saw  only  28%  of  the  respondents  contracting  for  this. 

Most  discouraging  is  the  continued  lack  of  contracted  support  in  the  area 
of  education  and  training.  A  majority  of  all  software  interruptions  can  be 
attributed  to  user  misuse  or  abuse  of  their  package.  Most  often,  this  can 
be  connected  to  insufficient  training  and  incomplete  (or  incomprehen- 
sible) documentation.  Software  support  vendors  could  improve  user 
satisfaction  while  at  the  same  time  reduce  support  costs  by  emphasizing 
training. 


Exhibit  n-2  acts  to  reinforce  the  findings  previously  presented  in  this  study: 

•  First,  that  phone  support  is  viewed  by  software  users  as  the  most  impor- 
tant form  of  support  that  they  receive  directly  from  their  vendors,  and, 
for  the  most  part,  vendor  performance  currentiy  satisfies  user  require- 
ments. 

•  Second,  that  software  documentation  does  not  even  come  close  to 
satisfying  the  needs  of  users. 

Users  have  assumed  much  responsibility  for  the  operations  and  support  of 
their  applications  software.  Because  of  this,  users  place  a  premium  on 
clearly  presented  documentation.  Unfortunately,  current  documentation 
often  is  poorly  written  and  presented.  This  shortcoming  is  exacerbated  by 
declining  experience  levels  common  at  user  organizations. 

This  dissatisfaction  with  the  documentation  acts  to  lessen  user  satisfac- 
tion with  other  components  of  the  support  and  usage  of  the  software 
product.  For  example,  users'  satisfaction  with  the  engineer  with  whom 
they  have  to  deal  as  a  result  of  not  understanding  the  documentation, 
places  a  strain  on  the  user-engineer  relationship,  particularly  if  the  solu- 
tion offered  by  the  support  specialists  is  to  "look  it  up  in  the  documenta- 
tion." 

The  value  that  users  place  upon  these  services  is  increasing  steadily  as 
users'  expectations  for  100%  system  availability  increases  and  advances 
in  hardware  reliability  place  the  focus  of  attention  on  the  software  com- 
ponents. 


Software  Support 
Performance 
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EXHIBIT  11-2 


SOFTWARE  DOCUMENTATION  NEEDS  UNMET 


VALUE 

A  =  On-Site  Support 

B  =  Phone  Support 

C  =  Documentation 

D  =  Software  Engineer  Skill  Level 
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c  

Documentation  Key      The  extent  of  user  dissatisfaction  with  documentation  is  illustrated  in 
Support  Deficiency       Exhibit  II-3,  which  shows  that  57%  of  the  users  surveyed  in  1987  were 

dissatisfied  with  their  documentadon.  This  reflects  a  decline  in  satisfac- 
tion from  1986,  when  54%  of  the  sample  reported  that  they  were  dissatis- 
fied with  their  documentation. 


EXHIBIT  11-3 


SOFTWARE  SUPPORT  SATISFACTION 
BY  SUPPORT  AREA 


SUPPORT 
AREA 

DISSATISFIED 

SATISFIED 

On-Site  Support 
Phone  Support 
Consulting 
Training 
Documentation 

1        1        1  1 

1       1       1  1 

100%    80      60      40      20       0       20      40      60      80  100% 


Percent  of  Sample 
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When  asked  what  single  change  or  improvement  the  users  would  like  to 
see,  documentation  was  far  and  away  the  number  one  choice,  specified 
by  47  out  of  the  302  users  (or  16%  of  the  sample)  surveyed.  No  other 
support  or  performance  feature  was  mentioned  more  than  ten  times. 

Most  often,  users  complained  about  the  lack  of  clarity  of  the  documenta- 
tion or  the  "user  friendliness"  of  the  manuals.  The  users  requested  better 
indexing  and  more  comprehensive  trouble- shooting  guides. 

In  general,  it  appears  that  documentation  rarely  is  written  with  the  audi- 
ence in  mind.  Often,  documentation  is  written  by  the  product  design 
team,  who  frequendy  make  assumptions  about  the  end  user's  level  of 
expertise  that  are  not  valid.  Lastly,  the  lack  of  effective  end-user  training 
further  highlights  the  deficiencies  of  software  documentation. 


Third-Party  Software    As  hardware  maintenance  price  reductions  decrease  future  hardware 
Support  Opportunities   maintenance  revenue  opportunities,  third-party  maintenance  (TPM) 

organizations  are  being  forced  to  consider  alternative  sources  of  reve- 
nues. Many  TPMs  have  identified  the  telecommunications  product 
marketplace  as  one  possible  "replacement,"  but  a  shortage  in  skilled 
technicians  and  expertise  has  acted  to  limit  growth  in  this  market. 

Instead,  some  TPMs  have  looked  to  increasing  their  involvement  in 
software  support  as  a  new  strategic  direction.  One  leading  vendor. 
Control  Data  Corporation,  has  drawn  from  its  experience  as  both  an 
equipment  vendor  and  a  major  user  by  offering  third-party  support  for 
both  DEC  and  IBM  software.  While  most  other  TPMs  have  cautiously 
taken  a  wait-and-see  attitude  (one  TPM  official  said  that  they  "let  CDC 
be  the  pioneer  that  catches  all  the  arrows"),  most  major  TPMs  are  at  least 
examining  the  software  support  market  for  future  entry. 

Exhibit  n-4  suggests  that  software  users  currentiy  show  limited  enthusi- 
asm for  third-party  software  support,  with  only  19%  of  the  sample  will- 
ing to  consider  such  an  alternative.  This  finding  is  also  supported  by 
earlier  research  performed  by  INPUT  which  found  that  only  24%  of 
current  TPM  users  would  be  willing  to  have  their  TPM  support  their 
applications  software. 

These  findings  are  more  a  reflection  of  user  perception  that  TPMs  do  not 
have  the  expertise  to  provide  such  support.  GE,  a  leader  in  the  TPM 
market,  was  recentiy  quoted  to  the  effect  that  TPMs  do  not  have  the  same 
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Willing  to  Consider 
Third-Party  Support 


level  of  expertise  as  the  software  vendors.  However,  TRW,  Sorbus 
(through  its  parent  Bell  Atlantic),  CDC,  GE,  and  Intelogic  Trace  all  have 
had  some  contact  with  software  as  a  supplier  and,  as  a  result,  should  draw 
upon  and  market  this  experience  in  order  to  exploit  this  market. 
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In  1987,  INPUT  surveyed  302  users  of  non-operating  systems  and  appli- 
cations products  from  ten  vendors:  ADR,  Cullinet,  DEC,  IBM, 
McCormick  &  Dodge,  MSA,  NCA,  Pansophic,  Sterling,  and  UCCEL. 
This  sample  size  represented  an  increase  of  almost  40%  over  1986,  when 
216  users  were  interviewed.  All  interviews  were  conducted  by  telephone, 
and  each  interview  lasted  approximately  twenty  minutes  in  length. 

On  the  average,  the  software  support  users  reported  that  they  paid  13.4% 
of  the  package's  license  for  software  support,  up  from  last  year's  average 
of  12.3%.  As  opposed  to  hardware  maintenance  respondents,  very  few 
software  users  mentioned  support  prices  as  a  problem  they  felt  needed 
attention,  giving  credence  to  INPUT'S  recommendation  that  vendors 
should  consider  price  increases  in  order  to  support  quality. 

Exhibit  ni-l  analyzes  the  software  support  delivery  modes  experienced 
by  the  software  user  sample.  Much  like  last  year,  the  1987  sample  re- 
ported greatest  experience  with  telephone  support  as  their  contractual 
support  offering,  with  66%  of  the  sample  contracting  for  this  offering 
versus  only  31%  who  contract  for  on-site  support.  Users  often  opt  for 
phone  support  due  to  its  responsiveness  and  lower  cost.  When  used  in 
conjunction  with  problems  data  bases,  phone  support  can  often  provide 
prompt  resolution  to  most  circumventable  problems. 

Interestingly,  almost  nine  out  of  ten  users  had  some  experience  with  on- 
site  support  on  an  ad  hoc  basis,  presumably  paying  on  a  hourly  basis  for 
such  support,  although  a  few  may  have  received  limited  on-site  support  at 
the  time  of  installation. 

Encouraging  is  the  growth  in  experience  with  remote  support,  contracted 
by  almost  50%  of  the  1987  sample  versus  39%  of  last  year's  user  base. 
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Such  usage  indicates  that  both  users  and  vendors  are  benefitting  from  the 
improved  support  and  reduced  costs  inherent  in  the  remote  down-line 
loading  of  software  fixes. 

Less  encouraging  is  the  lack  of  contractually- supplied  training  and 
consulting  support.  Less  than  two-thirds  of  the  users  reported  that  they 
received  satisfactory  levels  of  support  in  each  of  these  areas,  suggesting 
that  vendors  could  do  well  to  improve  the  performance  aspects  of  their 
support  offering  by  stressing  the  importance  of  these  service  areas  to  their 
users. 
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Exhibit  III-2  reports  on  the  number  of  major  and  minor  problems  encoun- 
tered by  the  sample  as  well  as  the  turnaround  time  required  to  resolve 
those  problems.  Major  problems  are  defined  as  those  that  prevent  further 
processing,  while  minor  problems  allow  continued  processing  with  some 
degradation. 


EXHIBIT  III-2 


SOFTWARE  SUPPORT  PERFORMANCE 

ALL  USERS 


SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.4 

Number  of  Major  Problems  Resolved  per  Month 

0.4 

Turnaround  Time  of  Major  Problems  (Hours) 

20.6 

Number  of  Minor  Problems  Reported  per  Month 

1 .5 

Number  of  Minor  Problems  Resolved  per  Month 

1.5 

Turnaround  Time  of  Minor  Problems  (Hours) 

26.5 

In  1987,  users  experienced  less  than  one  major  problem  per  every  two 
months,  which  is  down  slighdy  from  last  year's  results.  Turnaround  time 
on  those  problems  was  much  improved  in  1987,  averaging  20  hours 
versus  65  hours  in  1986.  Increased  usage  of  remote  support  and  reduced 
reliance  on  mailed  fixes  accounted  for  much  of  this  improvement. 


FUSW 


©  1987  by  INPUT.  Reproduction  Prohibited. 


15 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT 


INPUT 


The  users  encountered  fewer  minor  problems  in  1987,  1.5  per  month 
versus  2.6  in  1986.  Turnaround  time  was  reduced  from  an  obscenely 
high  200  hours  in  1986  to  a  much  more  responsive  26  hours  in  1987. 
Users  were  partly  to  credit  in  this  reduction  since  many  users  improved 
their  problem  tracking  and  documentation  procedures.  Moreover,  soft- 
ware vendors  began  addressing  minor  problems  in  a  much  more  respon- 
sive manner  instead  of  relying  on  scheduled  mailed  revisions  to  handle 
minor  software  "bugs." 

As  a  result,  user  satisfaction  with  both  major  and  minor  problem  resolu- 
tion was  quite  high  in  1987,  as  shown  in  Exhibit  111-3.  In  1986,  users 
rated  major  problem  resolution  an  8.2  and  major  problem  turnaround 
time  an  8.0,  versus  8.8  and  8.7  in  1987,  respectively.  Users  were  even 
more  pleased  with  minor  problem  resolution  and  turnaround  time,  rating 
both  resolution  and  turnaround  an  8.2  in  1987  (versus  7.9s  in  1986).  Not 
surprisingly,  overall  satisfaction  with  support  was  also  rated  higher,  8.2 
in  1987  versus  7.9  in  1986. 

Exhibits  ni-4  through  111-6  examine  software  support  performance  and 
user  satisfaction  in  greater  detail.  In  Exhibit  111-4,  users  rate  the  value 
placed  on  each  software  support  category,  at  the  same  time  rating  the 
performance  received  in  each  area.  Users  rated  documentation  as  their 
most  important  support  requirement,  followed  by  software  engineer  skill 
level  and  phone  support. 

Since  documentation  is  so  vital  to  the  end  user,  it  is  distressing  that 
documentation  performance  was  so  poorly  received.  Exhibit  111-5  shows 
that  only  43%  of  the  users  were  satisfied  by  their  documentation,  down 
slightiy  from  1986  when  46%  were  satisfied.  Documentation  clarity  and 
usability  was  most  often  cited  as  the  single  most  important  improvement 
desired  by  this  sample. 

Another  area  of  concern  is  training,  especially  in  light  of  user  dissatisfac- 
tion with  documentation.  Vendors  need  to  stress  the  need  for  contracted 
training  on  an  ongoing  basis. 

Users  also  expressed  some  level  of  dissatisfaction  with  the  skill  level 
demonstrated  by  the  software  support  engineer  skill  level.  In  part,  this 
dissatisfaction  may  be  directed  at  the  actual  expertise  of  the  engineer  in 
question.  More  likely,  the  users  were  dissatisfied  with  the  "people  skills" 
of  the  support  person  and  their  ability  to  communicate  with  this  person. 
In  any  case,  improvements  in  both  end-user  training  and  documentation 
will  act  to  improve  user  satisfaction  with  the  support  personnel. 
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EXHIBIT  III-4 


SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

ALL  USERS 


SOFTWARE 
bUrrUH 1 
CATEGORY 

LEVEL  OF  SUPPORT 

PERFORMANCE 

EXCEEDS 
(rALLo  bbLUW) 
USER  VALUE 

VALUt 

r tHrUnMANUt 

On-Site  Support 

5.7 

7.1 

1.4 

Phone  Support 

7.8 

8.3 

0.5 

Access  to  Problems 
Data  Base 

7.4 

7.1 

(0.3) 

Documentation 

8.7 

7.2 

(1.5) 

Training 

7.5 

6.7 

(0.8) 

Consulting 

7.2 

6.9 

(0.3) 

Software  Engineer 
Skill  Level 

8.3 

7.9 

(0.4) 

Scale:  1  =  Low,  10  =  High 

Average  Standard  Error  of  the  Mean:  0.2 


18 


©  1 987  by  INPUT.  Reproduction  Prohibited. 


FUSW 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT  INPUT 


EXHIBIT  III-5 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 
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Exhibit  III-6  graphically  plots  software  support  performance  versus  the 
value  that  users  place  on  each  service.  Again,  the  largest  gap  in  support 
performance  falls  in  the  areas  of  documentation  and  training. 


EXHIBIT  III-6 
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Exhibit  III-7  measures  software  user  willingness  to  consider  third-party 
software  support  as  an  alternative  to  their  current  software  vendor.  Only 
19%  of  the  software  support  user  sample  indicated  a  willingness  to  try 
third-party  software  support,  as  much  a  reflection  of  their  doubts  that 
such  support  is  actually  available  as  a  reflection  of  satisfaction  with  their 
current  support.  Indeed,  users  report  greatest  interest  in  third-party 
training  and  consulting,  two  areas  where  current  software  support  per- 
formance is  weakest. 
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Vendor  Performance  Analyses 


In  the  following  chapter,  INPUT  will  analyze  software  vendors'  perform- 
ances in  meeting  the  software  support  requirements  of  their  own  users  on 
a  vendor-by-vendor  basis.  Each  analysis  will  begin  with  a  description  of 
how  support  is  delivered  to  users.  Next,  each  vendor's  ability  to  deliver 
problem-free,  or  at  least  problem-resistant,  software  to  its  users,  as  well 
as  its  responsiveness  and  effectiveness  in  resolving  the  problem  is  exam- 
ined, followed  by  an  analysis  of  each  vendor's  performance  in  meeting 
the  requirements  that  users  have  for  specific  support  needs,  such  as  docu- 
mentation, training,  and  phone  support.  Lastly,  each  vendor's  user 
sample's  attitudes  towards  third-party  support  altematives  are  analyzed. 


ADR  INPUT  surveyed  thirty  users  of  ADR  software  in  1987.  The  sample  can 

be  broken  down  further  as  follows:  Seventeen  users  of  Librarian  (a  source 
program  management  system),  ten  users  of  Datacom/DB  (a  data  base/file 
management  system),  and  one  user  each  of  Empire  (a  decision  support 
system).  Look  (a  performance  measurement  system),  and  Vollie  (a 
programming  utility  package). 

These  packages  run  on  IBM  370  architecture  and  the  sample  hardware 
breakdown  reflects  this:  Thirteen  IBM  30XXs,  fifteen  IBM  43XXs,  and 
one  NAS  AS/8XXX. 

Most  of  the  software  programs  fall  in  the  $20,000-30,000  range,  with  the 
exception  of  Datacom/DB,  which  runs  between  $89,000-116,000.  On  the 
average,  users  paid  14.8%  of  the  license  fee  for  annual  software  support. 

Exhibit  rV-l  presents  a  picture  of  how  users  typically  receive  their  sup- 
port from  ADR  While  aU  ADR  users  receive  some  level  of  on-site 
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support,  only  37%  reported  that  they  receive  on-site  support  contractu- 
ally. Half  of  the  users  opt  for  telephone  support  contracts,  usually  in 
conjunction  with  access  to  a  problems  data  base  and  remote  support. 

ADR  offers  initial  training  free  of  charge  to  users;  after  that,  users  must 
pay  for  additional  training  on  a  per  person,  per  day  charge  that  runs 
between  $300-1000  for  three-  to  five-day  courses.  Half  of  the  users 
contract  for  training;  however,  over  one-third  of  the  users  reported  that 
they  receive  no  additional  training. 
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One  reason  that  so  few  ADR  users  contract  for  support  beyond  the  first 
year  of  free  coverage  may  be  the  reliability  of  the  packages,  shown  in 
Exhibit  IV-2.  ADR  users  reported  so  few  major  problems  that  it  was 
impossible  to  present  a  valid  problem  resolution  turnaround  time.  ADR 
software  was  also  relatively  free  from  minor  problems,  averaging  just 
over  one  per  month.  Problem  resolution  occurred  in  less  than  one  day, 
reflecting  the  effectiveness  of  telephone  and  remote  support  activities  at 
ADR. 


EXHIBIT  IV-2 


SOFTWARE  SUPPORT  PERFORMANCE 

ADR 


■ 

SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.1 

Number  of  Major  Problems  Resolved  per  Month 

0.1 

Turnaround  Time  of  Major  Problems  (Hours) 

* 

Number  of  Minor  Problems  Reported  per  Month 

1.1 

Number  of  Minor  Problems  Resolved  per  Month 

1.1 

Turnaround  Time  of  Minor  Problems  (Hours) 

6.1 

*insufficient  Response 


FUSW 


©  1987  by  INPUT.  Reproduction  Prohibited. 


25 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT 


INPUT 


Thus,  it  is  not  surprising  that  ADR  users  rated  their  software's  freedom 
from  problems  and  their  support  vendors'  responsiveness  and  effective- 
ness extremely  high  in  Exhibit  rV-3.  The  ADR  sample  rated  their  overall 
satisfaction  with  support  lowest  (8.1  out  of  10),  suggesting  that  while 
ADR  software  is  problem-free,  perhaps  the  support  delivery  may  have  a 
few  problems. 


EXHIBIT  IV-3 
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Exhibits  IV-4  through  IV-6  present  a  more  detailed  analysis  of  ADR  user 
satisfaction  with  ADR  service  performance.  Exhibit  IV-4  indicates  that 
ADR  users  place  the  highest  priority  on  documentation,  phone  support, 
and  access  to  problems  data  bases.  ADR  performance,  as  rated  by  the 
ADR  sample,  does  not  meet  the  mean  value  in  the  areas  of  documentation 
and  problems  data  bases. 

EXHIBIT  IV-4 
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EXHIBIT  IV-5 
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EXHIBIT  IV-6 
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EXHIBIT  IV-7 
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EXHIBIT  IV-8 
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Indeed,  ADR  performance  in  satisfying  individual  user  needs  is  very  poor 
in  the  problematic  area  of  documentation,  where  ADR  satisfied  only  13% 
of  its  users  in  1987,  down  significantly  from  1986  (when  ADR  met  the 
needs  of  50%  of  its  users).  Eleven  of  the  thirty  respondents  mentioned 
documentation  as  the  single  improvement  desired,  with  typical  comments 
such  as  "better  clarity,"  "more  complete  documentation,"  and  "more 
examples  in  the  documentation." 

In  1986,  ADR  users  reported  that  phone  support  was  problematic  (only 
47%  of  the  1986  sample  was  satisfied),  with  a  localized  group  of  users 
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suggesting  that  software  engineer  skill  level  needed  improvements.  In 
1987,  software  engineer  skill  level  received  much  higher  marks  (satisfy- 
ing all  of  the  ADR  sample),  and,  as  a  result,  ADR  user  satisfaction  with 
phone  support  also  rose. 

Still,  ADR  users  appear  very  happy  with  their  software  products,  and  this 
satisfaction  carries  over  into  the  support  areas.  Only  17%  of  the  sample 
indicated  willingness  to  try  third-party  support  for  their  software,  and 
most  of  that  willingness  centered  around  such  operations/performance 
support  activities  as  training  and  consulting.  Apparendy,  ADR  users  are 
willing  to  gamble  on  the  reliability  of  their  products  and  would  rather  act 
to  optimize  the  performance  on  an  as-needed  basis. 

B 


In  1987,  INPUT  interviewed  thirty  users  of  Cullinet  software,  broken 
down  by  product  as  follows:  Fifteen  users  of  IDMS  (a  communications 
control  data  base  management  system),  nine  users  of  IDD  (an  integrated 
data  dictionary),  three  users  of  CULPRIT  (a  report  generator),  two  users 
of  ADS/Online  (an  applications  development  tool),  and  one  user  of 
INTERACT  (a  text  editing  program). 

The  Cullinet  software  analyzed  ran  on  IBM  and  plug-compatible  main- 
frames, broken  down  as  follows:  Thirteen  IBM  303Xs,  eleven  43XXs, 
two  NAS  AS/SXXXs,  two  Amdahl  58XXs,  and  one  each  of  Amdahl  470 
and  Magnuson  M80. 

Most  of  the  software  packages  carried  license  fees  that  ran  between 
$27,000  (for  CULPRIT)  to  $60,000  (for  IDMS).  On  average,  CuUinet 
users  paid  12.9%  for  annual  support  fees;  however,  the  range  of  response 
was  fairly  broad  (2-40%). 

Exhibit  rV-l  shows  how  software  support  is  typically  delivered  to 
Cullinet  users.  Most  often,  Cullinet  users  receive  their  support  via  the 
telephone,  as  57%  of  the  sample  reported  that  they  contracted  for  this 
service.  Few  opt  for  on-site  support  and  even  less  contract  for  access  to  a 
problems  data  base  (in  fact,  only  57%  of  the  users  reported  any  contact 
with  such  a  data  base). 


Relatively  few  users  of  Cullinet  software  are  experienced  with  remote 
support  as  only  43%  of  the  sample  reported  that  they  receive  remote 
support  as  a  part  of  their  support  contract  and  37%  said  that  they  had  no 
experience  at  all  with  remote  support. 
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Exhibit  IV-9  shows  that  Cullinet  users  reported  that  they  experienced 
fewer  major  problems  than  the  entire  sample  (0.2  per  month  versus  0.4 
for  the  entire  sample),  as  well  as  fewer  minor  problems  (1.4  versus  1.5 
per  month).  The  Cullinet  sample  also  reported  that  turnaround  time  for 
major  problems  was  faster  than  the  entire  sample  (13.1  hours  versus  20.6 
for  the  entire  sample). 


EXHIBIT  IV-9 


SOFTWARE  SUPPORT  PERFORMANCE 

CULLINET 


 _____  

SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.2 

Number  of  Major  Problems  Resolved  per  Month 

0.2 

Turnaround  Time  of  Major  Problems  (Hours) 

13.1 

Number  of  Minor  Problems  Reported  per  Month 

1,4 

Number  of  Minor  Problems  Resolved  per  Month 

1.3 

Turnaround  Time  of  Minor  Problems  (Hours) 

32.6 

More  problematic  was  Cullinet  performance  in  resolving  minor  problems. 
The  users  reported  that  some  minor  problems  go  unsolved  (presumably 
resolved  by  mailed  revisions  sent  later).  The  relatively  long  turnaround 
also  indicates  an  area  of  concern,  suggesting  that  mailed  revisions  are 
frequentiy  relied  on  to  resolve  minor  "bugs."  The  danger  of  relying  on 
mailed  revisions  is  that  users  frequentiy  don't  realize  that  the  "bug"  has 
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has  been  rectified  (which  could  account  for  the  discrepancy  between 
minor  problems  reported  and  resolved),  even  though  the  vendor  docu- 
ments the  "improvements"  contained  in  the  revised  version  (often  in  the 
form  of  a  cover  letter,  but  also  in  the  actual  documentation).  This  contrib- 
utes to  user  dissatisfaction  with  both  the  support  and  the  product  itself. 


Cullinet  users  demonstrated  some  level  of  dissatisfaction  with  their 
software  package's  freedom  from  minor  problems  and  their  vendor's 
ability  to  overcome  those  problems  that  do  occur,  as  shown  in  Exhibit 
IV- 10.  More  importantly,  user  satisfaction  with  their  overall  support  was 
significantly  lower  than  the  entire  software  user  sample  (rated  7.5  on  a 
scale  of  10,  versus  8.2  for  the  entire  sample). 


EXHIBIT  IV-10 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 

CULLINET 
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Exhibits  IV- 1 1  through  IV- 13  take  a  closer  examination  of  Cullinet 
support  performance  as  reponed  by  the  1987  survey  group.  Exhibit  IV- 
1 1  indicates  that  Cullinet  users  place  the  greatest  value  on  their  software 
support  engineer's  skill  level,  phone  support,  and  documentation.  Exhibit 
rV-12  shows  that  Cullinet  manages  to  satisfy  the  majority  of  their  users' 
needs  in  each  of  these  areas;  however,  Cullinet  satisfies  at  least  two- 
thirds  of  its  customers  in  only  one  area  (phone  support). 

EXHIBIT  IV-11   ■  •  ' 

SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

CULLINET 


Ov-/r  1  vvMPiiz 
SUPPORT 
CATEGORY 

LEVEL  OF  SUPPORT 

PERFORMANCE 

L_  /\  O  L.  [_  O 

(FALLS  BELOW) 
USER  VALUE 

VALUE 

PERFORMANCE 

On-Site  Support 

4.8 

6.1 

1.3 

Phone  Support 

8.1 

7.9 

(0.2) 

Access  to  Problems 
Data  Base 

7.3 

8.0 

0.7 

Documentation 

7.9 

7.2 

(0.7) 

Training 

7.3 

6.6 

(0.7) 

Ccnsulting 

7.2 

6.8 

(0.4) 

Software  Engineer 
Skill  Level 

9.1 

8.3 

(0.8) 

Scale:  1  =  Low,  10  =  High 

Average  Standard  Error  of  the  Mean:  0.4 
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EXHIBIT  IV-12 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 

CULLINET 
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When  asked  what  single  improvement  Cullinet  users  desire  most,  users 
most  often  mentioned  documentation  and  software  support  personnel. 
Users  wanted  more  comprehensive  documentation  ("documentation  is  too 
vague")  and  better  communication  with  their  support  personnel. 
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Exhibit  IV-13  graphically  presents  the  discrepancy  between  user-reported 
value  assigned  to  phone  support,  documentation,  training,  consulting,  and 
software  engineer  skill  level  versus  the  actual  performance  reported  by 
Cullinet. 

EXHIBIT  IV-13 

SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 

CULLINET 


1        23456789  10 


VALUE 

A  =  On-Site  Support  E  =  Training 

B  =  Phone  Support  F  =  Consulting 

C  =  Access  to  Problems  Data  Base  G  =  Software  Engineer  Skill  Level 

D  =  Documentation 


FUSW 


e  1987  by  INPUT.  RaproducJlon  Prohibited. 


37 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT 


INPUT 


EXHIBIT  IV-14 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 
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Cullinet  users  were  among  the  most  willing  group  of  users  (third  after 
MSA  and  NCA)  with  regard  to  trying  third-party  software  support,  as 
shown  in  Exhibit  IV- 14.  Twenty-seven  percent  of  the  Cullinet  sample  in- 
dicated that  they  were  willing  to  consider  alternative  sources  for  their 
support.  Although  they  indicated  the  greatest  willingness  to  consider 
third-party  training,  user  willingness  was  fairly  evenly  spread  over  all  of 
the  support  services  analyzed. 

c  

DEC  Along  with  being  a  major  vendor  of  computer  equipment,  Digital  Equip- 

ment Corporation  (DEC)  is  a  leading  vendor  of  software  packages  for 
those  computer  systems.  In  1987,  INPUT  surveyed  thirty-one  users  of 
DEC  software  concerning  the  support  that  they  received.  All  of  these 
surveys  were  done  with  users  of  DEC's  All-in- 1,  an  integrated  office 
automation  program  designed  to  run  on  any  VAX  hardware. 

The  DEC  sample  can  be  further  broken  by  processor  as  follows:  Fifteen 
DEC  VAX/78XS,  eight  DEC  VAX/75Xs,  and  the  remaining  eight  split 
evenly  between  the  new  VAX  8XXXs  and  Micro  VAX  lis. 

DEC  offers  many  different  levels  of  support  for  its  software,  just  as  it 
does  for  its  hardware  systems.  Most  users  choose  Basic  Service,  which 
provides  telephone  support,  access  to  DEC's  problems  data  base  (called 
DSIN,  Digital's  Software  Information  Network),  media  and  documenta- 
tion updates,  and  technical  newsletters.  DEC  also  offers  a  premium 
service  level,  called  DEC  Support  Service,  which  adds  on-site  and  remote 
installations  of  updates,  preventive  maintenance,  remote  remedial  mainte- 
nance, and  on-site  remedial  support  during  normal  business  hours  when 
necessary. 

The  1987  All-in- 1  sample  reported  that  they  paid  approximately  20%  for 
annual  support,  suggesting  that  they  opted  for  the  DEC  Support  Service 
level  versus  Basic,  which  typically  runs  around  14%  annually. 

Exhibit  IV- 15  shows  that  42%  of  the  DEC  sample  has  had  to  receive 
support  on-site  during  contracted  hours  and  another  45%  experienced  on- 
site  support  on  an  ad  hoc  (billed  additionally)  basis.  Still,  most  software 
support  is  delivered  via  the  telephone. 

The  All-in- 1  users  also  made  use  of  remote  support  (84%)  and  the  DSIN 
problems  data  base.  On  the  other  hand,  few  DEC  users,  like  most  other 
software  users,  contracted  for  additional  training  or  consulting  services. 
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EXHIBIT  IV-15 


SOFTWARE  SUPPORT  DELIVERY 
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The  emphasis  on  remote  support,  along  with  DEC'S  excellent  support 
coverage  (via  the  telephone  support  center  in  Colorado  Springs,  CO), 
resulted  in  extremely  prompt  problem  resolution  (shown  in  Exhibit  IV- 
16),  particularly  for  major  software  problems.  Exhibit  IV-17  indicates 
that  All-in- 1  users  were  extremely  satisfied  with  their  software's  freedom 
from  major  problems  (rated  9.7  on  a  scale  of  10),  as  well  as  problem 
resolution  and  turnaround  time  (rated  9.4  and  9.5,  respectively). 


DEC  was  less  successful  in  satisfying  its  users  with  respect  to  minor 
problems,  specifically  freedom  from  problems  and  minor  problem  resolu- 
tion. 
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EXHIBIT  IV-16 


SOFTWARE  SUPPORT  PERFORMANCE 

DEC 
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SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.3 
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Number  of  Minor  Problems  Reported  per  Month 

3.8 

Number  of  Minor  Problems  Resolved  per  Month 

3.2 

Turnaround  Time  of  Minor  Problems  (Hours) 

12.3 

User  satisfaction  with  overall  software  support  is  rated  highly,  yet  user 
satisfaction  with  specific  support  services  is  significantly  lower,  as  shown 
in  Exhibits  IV- 18  through  IV-20.  Exhibit  IV- 18  indicates  that  DEC  users 
have  high  support  expectations  for  all  areas  of  support,  perhaps  a  result  of 
DEC'S  reputation  and  the  marketing  of  its  support  services.  Thus,  it 
becomes  more  difficult  to  meet  the  high  expectations  of  its  users,  as 
indicated  in  Exhibit  IV- 19. 

When  asked  to  elaborate  on  the  poor  ratings  given  to  DEC  software  support 
performance,  the  All-in- 1  users  responded  with  perceived  needs: 

•  "Better  documentation"  -  six  respondents. 

•  "Faster  responses"  -  three  respondents. 


FUSW 


©  1987  by  INPUT,  Reproduction  Prohibited. 


41 


USER  SERVICE  REQUIREMENTS-SOFTWARE  SUPPORT 


INPUT 


EXHIBIT  IV-17 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 
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•  "Improved  performance  of  the  software"  -  three  respondents. 

•  "Better  training;  too  complicated"  -  two  respondents. 

Still,  these  users  are  extremely  loyal  to  their  software  vendor,  as  evident 
in  Exhibit  IV-21,  which  shows  that  although  DEC  users  report  that  DEC 
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EXHIBIT  IV-18 


SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

DEC 
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Scale:  1  =  Low,  10  =  High 

Average  Standard  Error  of  the  Mean:  0.2 


does  not  meet  any  of  their  other  extremely  high  support  needs,  none  of 
the  All-in- 1  users  were  willing  to  consider  third-party  sources  as  an 
alternative  to  their  current  support  vendor. 
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EXHIBIT  IV-19 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 

DEC 
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EXHIBIT  IV-20 


SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 
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EXHIBIT  IV-21 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 
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IBM  In  1987,  INPUT  interviewed  thirty  users  of  the  following  IBM  software 

products:  twenty-six  users  of  Manufacturing  Accounting  and  Production 
Information  Control  (MAPICS),  three  users  of  Distributed  Office  Support 
System  (DISSOS),  and  one  user  of  Communications  Oriented  Production 
Information  and  Control  System  (COPICS).  The  sample  can  also  be 
broken  down  by  the  hardware  that  the  software  ran  on:  twenty  System/ 
36s,  four  System/34s,  two  System/38s,  two  308Xs,  one  43XX,  and  one 
309X. 

MAPICS  is  a  modular  manufacturing  software  system  made  up  of  at  least 
thirteen  separate  but  integrated  m.odules  that  cost  between  $1,650  and 
$6,100  (one-time  charge).  COPICS  is  also  modular  in  design,  with 
modules  priced  between  $2,375  and  $20,480.  A  core  system  DISOSS 
runs  between  $29,000-$3 1,000  (one-time  charge).  The  sample  reported 
that  they  paid  an  average  of  14.5%  annually  for  support. 

As  a  company,  IBM  is  well  known  for  its  support,  a  factor  that  leads 
many  users  to  IBM  in  the  first  place.  Exhibit  IV-22  presents  the  IBM 
sample's  experience  with  various  software  support  services  and  the 
company's  delivery  methodology.  As  with  most  other  software  vendors, 
IBM  provides  most  support,  contractually  and  on  an  ad  hoc  basis,  via 
telephone  support,  usually  in  conjunction  with  a  problems  data  base. 
While  two-thirds  of  the  users  did  not  contract  for  on-site  support,  virtu- 
ally all  of  the  users  reported  receiving  on-site  support  at  some  level. 

Relatively  few  users  reported  that  they  contract  for  either  consulting  or 
training  services,  yet  most  users  reported  some  exposure  to  these  service 
on  an  ad  hoc  basis. 

Exhibit  IV-23  shows  that  the  IBM  software  was  extremely  reliable, 
averaging  only  one  major  problem  every  three  months  and  just  over  one 
minor  problem  per  month.  Turnaround  time  on  major  problems  took 
longer  than  on  minor  problems  (20.2  hours  versus  17.2  hours);  users  rated 
their  satisfaction  (in  Exhibit  IV-24)  witii  major  problem  resolution  turn- 
around time  lower  than  with  minor  problem  resolution  turnaround  time 
(8.6  on  a  scale  of  10  compared  to  8.9). 
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EXHIBIT  IV-22 


SOFTWARE  SUPPORT  DELIVERY 

IBM 
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EXHIBIT  IV-23 


SOFTWARE  SUPPORT  PERFORMANCE 

IBM 


 —  ■  —  

SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.3 

Number  of  Major  Problems  Resolved  per  Month 

0.3 

Turnaround  Time  of  Major  Problems  (Hours) 

20.2 

Number  of  Minor  Problems  Reported  per  Month 

1,1 

Number  of  Minor  Problems  Resolved  per  Month 

1.1 

Turnaround  Time  of  Minor  Problems  (Hours) 

17.2 
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EXHIBIT  IV-24 


USER  SATISFACTION  WITH 
SOFTWARE  SUPPORT  PERFORMANCE 

IBM 
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Still,  IBM  users  reported  high  satisfaction  with  their  software  package's 
reliability  and  the  quality  of  support  received.  Exhibit  IV-25  breaks 
down  IBM  software  support  performance  by  service  component,  indicat- 
ing that  IBM  performance  exceeds  the  value  that  its  users  assigned  to 
each  support  service  analyzed.  IBM  users  placed  the  highest  importance 
on  documentation  and  software  engineer  skill  level,  and  IBM  succeeded 


EXHIBIT  IV-25 

SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

IBM 
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in  satisfying  its  users  even  in  these  key  areas.  Exhibit  IV-26  shows  that 
IBM  satisfied  59%  of  the  IBM  user  sample's  documentation  needs  and 
77%  of  the  users'  requirements  regarding  software  engineer  skill  level 
(compared  to  43%  and  64%  for  the  entire  software  user  sample).  Only 
four  of  the  thirty  IBM  users  surveyed  mentioned  documentation  as  the 
single  improvement  desired. 


EXHIBIT  IV-26 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 

IBM 
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IBM  is  successful  at  satisfying  its  users'  needs  due  to  its  ability  to  deter- 
mine the  specific  support  needs  of  its  users  and  then  provide  support  at 
that  level.  Exhibits  rV-25  and  IV-27  show  that  IBM  users  reponed  lower 
support  requirements  than  other  product  groups,  and  in  many  cases,  other 
support  vendors  provide  better  support  (on  an  absolute  basis),  yet  IBM 
usually  receives  better  support  satisfaction  marks  than  most  vendors  due 
to  this  ability  to  provide  just  the  right  amount  of  support. 

Even  though  IBM  software  support  performance  satisfies  most  users' 
needs,  Exhibit  VI-28  shows  that  23%  of  the  IBM  sample  were  willing  to 
consider  a  third-party  source  of  software  support,  which  is  slightly  higher 
than  the  software  user  sample  on  the  whole.  This  may  be  due  to  the  large 
percentage  of  IBM  respondents  who  used  System/3Xs,  which  are  under 
increased  competition  from  third-party  maintenance  firms  that  are  vying 
for  the  hardware  maintenance  business.  This  increased  pressure  from 
TPMs  may  have  increased  the  visibility  of  alternative  support  sources  in 
general.  Nonetheless,  IBM  users  seem  quite  willing  to  consider  alterna- 
tive training  sources. 


McCormack  &  McCormack  &  Dodge  is  a  leading  vendor  of  accounting  software  for  the 

Dodge  IBM  mainframe  environment.  In  1987,  INPUT  interviewed  thirty  users 

of  the  following  M&D  software  (in  some  combination):  fifteen  users  of 
A/P  Plus  (an  accounts  payable  package),  12  users  of  GLiMillennium  (a 
general  ledger  package),  ten  users  of  P/A  Plus  (a  fixed  assets  package), 
four  users  of  A/R  Plus  (a  payroll  reporting  package),  and  two  users  of 
HRiMillennium  (a  personnel  management  package). 

The  sample  can  be  further  broken  down  into  the  following  machine  types: 
Thirteen  IBM  30XXs,  twelve  IBM  43XXs,  two  Amdahl  470s,  two  Hon- 
eywell DPS8s,  and  one  NAS  AS/8XXX. 

The  McCormack  &  Dodge  software  analyzed  is  priced  from  $10,000  to 
$40,000  per  package,  and  the  average  support  price  reported  by  the 
sample  was  12.7%  of  the  license  fee. 

McCormack  &  Dodge  emphasizes  its  training  services,  which  include 
both  data  processing  and  accounting  curriculum.  In  addition,  McCor- 
mack &  Dodge  offers  computer-based  training  on  selected  accounting 
packages. 
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EXHIBIT  IV-27 


SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 

IBM 


VALUE 

A  =  On-Site  Support  E  =  Training 

B  =  Phone  Support  F  =  Consulting 

C  =  Access  to  Problems  Data  Base  G  =  Software  Engineer  Skill  Level 

D  =  Documentation 
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EXHIBIT  IV-28 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 

IBM 
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It  is  apparent  from  Exhibit  IV-29  that  few  users  contract  for  support  from 
McCormack  &  Dodge  since  only  27%  of  the  sample  reported  that  they 
receive  on-site  support  contractually  and  47%  reported  that  they  contract 
for  phone  support  (although  all  of  the  users  reported  that  they  had  some 
experience  with  on-site  and  phone  suppon  on  an  ad  hoc  basis).  What 
support  services  the  users  did  contract  for  were  training  (57%  of  the 
sample)  and  access  to  problems  data  bases  (60%). 


EXHIBIT  IV-29 
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Exhibit  IV-30  attests  to  the  commendable  reliability  of  the  McCormack  & 
Dodge  software  products  analyzed.  The  user  sample  reported  only  one 
major  problem  every  five  months,  so  few  that  it  was  impossible  to  meas- 
ure average  turnaround  on  major  problem  resolution.  The  users  also 
reported  that  their  software  was  relatively  free  from  minor  problems, 
although  some  problems  appear  to  go  unresolved  (presumably  resolved  in 
a  later  revision). 


EXHIBIT  IV-30 

SOFTWARE  SUPPORT  PERFORMANCE 
McCORMACK  &  DODGE 


SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.2 

Number  of  Major  Problems  Resolved  per  Month 

0.2 

Turnaround  Time  of  Major  Problems  (Hours) 

* 

Number  of  Minor  Problems  Reported  per  Month 

1.2 

Number  of  Minor  Problems  Resolved  per  Month 

1.0 

Turnaround  Time  of  Minor  Problems  (Hours) 

28.1 

*lnsufficient  Response 


The  McCormick  &  Dodge  users  were  very  satisfied  with  their  software 
and  support  with  regard  to  major  problems,  as  shown  in  Exhibit  IV-31, 
rating  their  satisfaction  with  the  software's  freedom  from  major  problems 
a  9.5  on  a  scale  of  10  and  rating  both  major  problem  resolution  and 
turnaround  time  a  9.4. 
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EXHIBIT  IV-31 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 

McCORMACK  &  DODGE 
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These  users  were  less  satisfied  with  regard  to  minor  problems,  and  this 
tended  to  affect  their  overall  satisfaction  with  support,  which  received  a 
rating  of  only  7.7. 


Documentation  appears  to  be  a  major  contributor  to  any  dissatisfaction 
users  have  with  support,  as  shown  in  Exhibits  IV-32  through  IV-34. 
Exhibit  IV-32  indicates  that  McCormack  &  Dodge  users  place  the  great- 
est importance  on  documentation  (valued  at  a  9.5  on  a  scale  of  10). 
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Unfortunately,  these  users  rated  M&D  performance  a  6.0  on  the  same 
scale,  and,  as  a  result,  McCormack  &  Dodge  managed  to  satisfy  only 
17%  of  its  users  (Exhibit  IV-  33). 


EXHIBIT  IV-32 
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EXHIBIT  IV-33 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 
McCORMACK  &  DODGE 
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EXHIBIT  IV-34 


SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 
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EXHIBIT  IV-35 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 
McCORMACK  &  DODGE 
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Beyond  documentation,  McCormack  &  Dodge  support  performance 
manages  to  satisfy  the  majority  of  its  users'  needs,  although  it  appears 
that  a  small  group  of  extremely  dissatisfied  users  are  skewing  "perform- 
ance" ratings  in  Exhibits  IV-32  and  rV-34.  This  accounts  for  the  fact  that 
the  majority  of  users  are  satisfied  with  consulting  and  access  to  problems 
data  bases  even  though  the  users'  performance  ratings  fall  short  of  their 
value  ratings. 

Exhibit  IV-35  indicates  20%  of  the  McCormack  &  Dodge  users  were 
willing  to  consider  a  third-party  source  of  software  support.  These  users 
rated  their  willingness  to  use  third-party  support  in  the  areas  of  consulting 
and  training  very  highly,  suggesting  a  good  opportunity  for  third-party 
penetration  into  this  market. 

F  

MSA  ^  1987,  INPUT  surveyed  thirty  users  of  Management  Science  America 

(MSA)  accounting  software  products  that  were  typically  used  in  some 
combination  by  all  users  in  the  MSA  sample. 

The  MSA  sample  typically  used  IBM  mainframes,  predominantly  IBM 
43XXs  and  308X/9Xs,  although  two  users  used  NAS  AS  mainframes  and 
one  used  a  Honeywell  mainframe  system. 

Depending  upon  the  module  selected  and  the  features  contained  in  each 
program,  MSA  accounting  packages  can  range  in  price  from  $25,000  up 
to  $150,000.  MSA  reports  in  its  Form  10-K  that  support  charges  range 
from  8  to  12%  of  the  license  fee;  however,  our  sample  reported  that  they 
paid  between  7  and  30%,  averaging  14%  of  the  license. 

Exhibit  IV-36  presents  a  picture  of  MSA  support  delivery  as  reported  by 
its  users.  While  MSA  makes  on-site  support  available  to  its  end  users,  it 
is  apparent  that  most  users  (79%  of  the  sample)  receive  their  support  via 
the  telephone.  Users  who  purchase  MSA's  Expertlnsight  module  of 
ExpertLink  program  can  access  a  problems  data  base  directly,  and  55%  of 
this  sample  went  this  route.  ExpertLink  also  provides  a  facility  to  remote 
support,  and  two-thirds  of  the  sample  take  advantage  of  this  capability. 

MSA  stresses  its  involvement  in  the  area  of  training,  offering  users 
training  credits  that  vary  by  product  Forty-one  percent  of  the  MSA  users 
contract  for  training,  another  55%  receive  their  training  via  the  credits 
provided  with  their  software  purchase. 
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EXHIBIT  IV-36 


SOFTWARE  SUPPORT  DELIVERY 
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MSA  software  reliability  appears  to  be  below  average  in  terms  of  free- 
dom from  major  and  minor  problems,  as  reported  by  MSA  users  in 
Exhibit  IV-37.  Major  problem  resolution  is  significantly  better  than  the 
overall  sample's  responses  (14.1  hours  versus  20.6  hours);  however, 
minor  problem  resolution  time  is  only  slightly  better  than  the  entire 
sample's. 
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EXHIBIT  IV-37 


SOFTWARE  SUPPORT  PERFORMANCE 

MSA 


 —  ,  __  

Number  of  Maior  Problems  Reoorted  oer  Month 

n  Q 

Number  of  Maior  Problem*?  Rp<^ol\/pd  ner  Month 

n  Q 

Turnaround  Time  of  Major  Problems  (Hours) 

14.1 

Number  of  Minor  Problems  Reported  per  Month 

2.2 

Number  of  Minor  Problems  Resolved  per  Month 

2.2 

Turnaround  Time  of  Minor  Problems  (Hours) 

24.3 

The  MSA  users  rated  their  satisfaction  with  each  of  these  software  sup- 
port criteria  significantly  lower  than  the  entire  sample  (see  Exhibit  IV- 
38),  suggesting  that  MSA  users  are  expecting  more  from  their  support 
vendor.  Upon  closer  examination  of  performance  and  value  levels  in 
Exhibit  rV-39,  it  can  be  seen  that  MSA  users  relatively  are  pleased  with 
certain  aspects  of  their  support,  namely  on-site  support,  problems  data 
bases,  and  training  services  received,  yet  concemed  with  areas  most 
valuable  to  them: 

•  Documentation,  valued  at  a  9.0  (on  a  scale  of  10),  rated  only  a  7.3  in 
performance  received  (shown  in  Exhibit  rV-39). 

•  Software  engineer  skill  level,  valued  a  9.3,  rated  a  7.2  in  performance 
received. 
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EXHIBIT  IV-38 


USER  SATISFACTION 
WITH  SOFTWARE  SUPPORT  PERFORMANCE 
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EXHIBIT  IV-39 


SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 
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Exhibit  IV-40  indicates  that  MSA  users  are  least  satisfied  with  these 
valued  services  as  MSA  manages  to  satisfy  only  37%  of  its  users'  docu- 
mentation needs  and  32%  of  the  users'  requirements  regarding  the  soft- 
ware engineer's  skill  level. 


EXHIBIT  IV-40 
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The  MSA  users  criticized  the  documentation's  clarity  and  reported  that 
their  documentation  was  frequendy  out-of-date.  The  users  also  stated 
that  response  time  was  a  problem. 

Exhibit  IV-41  graphically  presents  the  discrepancy  between  user-per- 
ceived value  of  these  services  versus  reported  service  performance.  MSA 
should  be  particularly  concerned  about  user  dissatisfaction  with  docu- 
mentation, which  was  also  the  biggest  problem  in  the  1986  MSA  per- 
formance analysis. 

This  building  dissatisfaction  may  cause  some  users  to  look  to  other 
sources  of  software  support.  Exhibit  IV-42  indicates  that  43%  of  the 
MSA  users  are  willing  to  consider  a  third-party  source  of  software  sup- 
port, particularly  in  the  areas  of  consulting  and  training.  This  willingness 
is  the  largest  of  all  products  analyzed. 


G  

NCA  NCA  is  a  leading  vendor  of  integrated  software  packages  that  combine 

manufacturing  and  financial  functions  in  powerful  applications  systems 
for  DEC  YAXP/MS  computer  systems. 

NCA  MAXCIM  Financial  System  is  composed  of  a  number  of  account- 
ing modules,  including  order  management,  general  ledger,  accounts 
payable  and  receivable,  and  fixed  assets,  to  name  a  few.  Depending  on 
the  number  of  modules  selected,  the  price  of  this  software  ranges  between 
$30,000  and  $128,000. 

NCA  MAXCIM  Manufacturing  System  combines  a  number  of  manufac- 
turing control  and  accounting  functions,  including  purchasing,  capacity 
planning,  inventory,  and  payroll.  This  package  costs  $50,000-$  100,000 
(one-time  license),  depending  on  modules  selected. 

NCA  charges  12%  per  year  for  support,  although  some  users  must  have 
negotiated  for  different  levels  of  support  since  the  responses  ranged  from 
10-15%. 
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EXHIBIT  IV-41 
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Exhibit  rV-43  indicates  that  the  vast  majority  of  NCA  users  opt  for 
telephone  suppon,  supplemented  (at  newer  installations)  by  remote 
support  (down-line  loading  of  fixes).  NCA  has  only  recently  begun 
offering  remote  support,  hoping  to  improve  support  responsiveness  and 
efficiency  while  at  the  same  time  (eventually)  reducing  support  costs. 

Thirty  percent  of  the  sample  also  opted  for  on-site  support  contracts,  which 
are  typically  billed  on  a  hourly  basis. 


Few  users  opted  for  additional  training  (23%  of  the  sample)  and  consult- 
ing (17%),  with  most  users  relying  on  the  initial  or  other  ad  hoc  support 
that  they  receive  in  these  areas. 


EXHIBIT  IV-43 
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Exhibit  IV-44  shows  that  NCA  users  reported  that  its  software  suffers  in 
its  freedom  from  both  major  problems  (at  1.4  per  month)  and  minor 
problems  (2.0  per  month),  both  areas  falling  behind  industry-wide  per- 
formance shown  in  Chapter  HI.  Furthermore,  the  users  reported  that 
some  major  problems  do  not  even  get  resolved  (1.4  problems  reported 
versus  1.2  resolved).  This  discrepancy  may  be  attributed  to  the  move 
towards  remote  support,  since  some  fixes  to  problems  observed  by  users 
and  resolved  remotely  may  not  always  be  perceived  by  the  user.  This  can 
be  avoided  by  providing  an  up-to-date  log  of  all  remote  support  activities. 

EXHIBIT  IV-44 


SOFTWARE  SUPPORT  PERFORMANCE 

NCA 


SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

1.4 

Number  of  Major  Problems  Resolved  per  Month 

1.2 

Turnaround  Time  of  Major  Problems  (Hours) 

29.2 

Number  of  Minor  Problems  Reported  per  Month 

2.0 

Number  of  Minor  Problems  Resolved  per  Month 

2.0 

Turnaround  Time  of  Minor  Problems  (Hours) 

14.0 

Turnaround  time  for  major  problems  is  an  even  greater  problem,  averag- 
ing almost  thirty  hours  compared  to  twenty-one  hours  for  the  entire 
software  sample.  This  seems  excessively  high,  considering  the  move  to 
remote  support.  NCA  performance  is  much  better  for  minor  problems, 
which  required  fourteen-hour  turnaround. 
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Exhibit  IY-45  reflects  lower  than  average  user  satisfaction  with  NCA 
reliability  and  support  responsiveness,  with  user  ratings  ranging  between 
6.8  and  7.4  (on  a  scale  of  10),  versus  overall  sample  ratings  between  8.2 
and  8.9. 


EXHIBIT  IV-45 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 
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NCA  users  place  the  greatest  importance  on  the  support  organization's 
software  engineer  skill  level  (valued  as  a  8.0  on  a  scale  of  10)  and  the 
documentation  (valued  as  a  7.8).  Unfortunately,  NCA  performance  is 
rated  (in  Exhibit  IV-46)  significandy  lower  in  these  two  areas,  7.0  and 
6.4,  respectively. 
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EXHIBIT  IV-46 


SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 
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These  lower  performance  ratings  may  be  partly  caused  by  a  small  group 
of  extremely  displeased  users  whose  responses  may  have  skewed  the 
averages,  but  Exhibit  IV-47  shows  that  NCA  still  only  satisfies  less  than 
55%  of  its  users  in  its  two  important  areas. 


EXHIBIT  IV-47 
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When  asked  what  single  improvement  users  desired  most,  six  users 
pointed  to  the  number  of  skilled  support  personnel  available.  Specific 
responses  included: 

•  "Be  more  definite  on  support  decisions." 

•  "Better  knowledge  of  the  bugs." 

•  "Raise  the  level  of  technical  support." 

Five  users  specified  documentation  improvements  as  the  most  important 
improvement  desired. 

Exhibit  IV-48  graphically  illustrates  the  gap  between  user-perceived  value 
and  vendor  performance  in  these  areas. 

Not  surprisingly,  a  relatively  large  percentage  of  the  NCA  users  were 
willing  to  consider  a  third-party  source  of  support.  Exhibit  IV-49  shows 
that  30%  of  the  NCA  sample  would  consider  an  alternative  source  of 
support,  particularly  an  altemative  source  of  phone  support. 

H  

Pansophic  In  1987,  INPUT  surveyed  tiiirty  users  of  Pansophic  Systems  software 

products,  including  thirteen  users  of  Easytrieve,  a  database  file  manage- 
ment system;  ten  users  of  Panvalet,  a  library  management  system;  four 
users  of  Gener/OL,  an  application  development  tool;  and  three  users  of 
O-W-L,  a  programming  utility. 

Pansophic  prices  its  software  as  follows:  Panvalet,  $10,000-  $22,000; 
O-W-L,  $20,000-$27,000;  Easytrieve,  $21,000-$27,000;  and  Gener/OL, 
$35,000-$40,000.  Pansophic  users  reported  that  they  paid  15%  of  the 
software  license  fee  per  year  for  support. 

Exhibit  IV-50  shows  that  the  majority  of  contractual  software  support  is 
delivered  by  telephone,  although  40%  of  :he  users  reported  that  they 
receive  on-site  support  as  a  part  of  their  contract. 

In  1986,  Pansophic  stated  a  desire  to  increase  its  involvement  in  remote 
support  technology,  and,  in  1987,  45%  of  its  sample  reported  receiving 
this  service  contractually;  anodier  25%  received  it  on  an  ad  hoc  basis. 
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EXHIBIT  IV-48 


SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 
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EXHIBIT  IV-49 
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THIRD-PARTY  SOFTWARE  SUPPORT 
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EXHIBIT  IV-50 


SOFTWARE  SUPPORT  DELIVERY 
PANSOPHIC 
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The  Pansophic  sample  reponed  in  Exhibit  rV-5 1  that  their  software  rarely 
suffered  any  major  or  minor  problems,  averaging  only  one  problem  every 
ten  months.  As  a  result,  it  was  impossible  to  get  a  valid  turnaround  time. 
Exhibit  rV-52  demonstrates  that  Pansophic  users  are  extremely  pleased 
with  their  software,  both  in  its  ability  to  avoid  major  and  minor  problems 
and  in  Pansophic 's  support  performance  in  the  event  of  a  problem. 


EXHIBIT  IV-51 

SOFTWARE  SUPPORT  PERFORMANCE 
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EXHIBIT  IV-52 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 
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The  reliability  of  the  software  has  apparently  reduced  the  value  of  sup- 
port in  the  eyes  of  Pansophic  users;  Exhibit  rV-53  shows  that  Pansophic 
users  place  a  high  value  on  only  one  support  area-documentation. 


EXHIBIT  IV-53 
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Since  Pansophic  users  do  not  place  a  premium  on  support  (one  may  argue 
that  its  software  rarely  requires  support),  it  is  fairly  easy  for  Pansophic  to 
meet  the  needs  of  its  users.  Exhibit  rV-54  indicates  that  a  large  percent- 
age of  Pansophic  users  are  satisfied  with  their  suppon,  the  lowest  percent- 
age of  user  satisfaction  being  for  training  (54%  satisfied). 


EXHIBIT  IV-54 
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Nine  users  stated  that  they  were  very  pleased  and  offered  no  suggestion 
for  improvements.  Two  users  desired  better  documentation,  and  one  user 
mentioned  that  the  turnaround  time  could  be  improved. 

Still,  user  satisfaction  was  extremely  high,  particularly  in  the  areas  of 
software  engineer  skill  level  (86%  satisfied)  and  phone  support  (90% 
satisfied).  Pansophic  received  even  higher  marks  in  other  support  areas, 
such  as  on-site  support.  Of  course,  user  requirements  for  these  services 
are  lower  (as  shown  in  Exhibit  IV-  55),  making  it  easier  to  satisfy  the 
users'  needs. 

Not  surprisingly,  few  Pansophic  users  indicated  that  they  would  be 
willing  to  consider  an  alternative  source  of  support,  as  shown  in  Exhibit 
IV-56. 

I  

Sterling  Software  Sterling  Software  is  really  a  collective  of  six  different  operating  groups 

(the  Systems  Software  Group,  the  Federal  Systems  Group,  the  Profes- 
sional Services  Group,  the  Information  Systems  Group,  the  Financial 
Software  Group,  and  the  Business  Management  Group)  that  act  as 
wholly-  owned  subsidiaries.  Since  all  of  Sterling's  software  efforts 
resulted  from  acquisition,  Sterling  allows  each  operating  group  to  set 
support  policies  as  they  see  best. 

In  1987,  INPUT  surveyed  users  of  three  Sterling  packages  out  of  the 
Systems  Software  Group,  broken  down  as  follows:  Twenty-three  users  of 
DMS/OS,  a  DASD  management  system,  and  eight  users  of  TRACS  and 
SUPERTRACS  teleprocessing  remote  access  control  systems. 

These  products  were  designed  to  run  on  IBM  mainframe-type  products 
and  the  sample  reflects  this:  thirteen  IBM  30XX  users,  ten  IBM  43XX 
users,  and  one  NAS  AS/8XXX  user. 

The  Sterling  software  analyzed  in  this  report  is  priced  as  follows:  DMS/ 
OS,  $14,000;  TRACS,  $7,000-$9,000;  and  SUPERTRACS,  $12,000- 
$15,000.  The  Sterling  users  reported  that  they  paid  12.7%  per  year  on 
support. 
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EXHIBIT  IV-55 
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EXHIBIT  IV-56 
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Exhibit  IV-57  shows  that  the  majority  of  Sterling  users  rely  on  telephone 
support  as  their  principal  source  of  contracted  support;  less  than  one-half 
of  the  sample  reported  any  contact  with  on-site  support;  even  fewer 
receive  their  support  contractually  in  this  manner.  In  fact,  the  Sterling 
sample  had  the  largest  percentage  of  users  of  all  product  samples  that 
reported  no  support  in  a  number  of  service  areas,  such  as  29%  who  have 
not  experienced  remote  support,  26%  who  have  not  experienced  training, 
and  35%  who  have  not  experienced  consulting  support. 


EXHIBIT  IV-57 
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The  Sterling  users  reponed  relatively  little  experience  with  software 
problems,  with  only  0.4  major  problems  and  1 .2  minor  problems  per 
month  (shown  in  Exhibit  IV-58).  User  satisfaction  is  quite  high,  as 
indicated  in  Exhibit  IY-59.  Users  rated  all  areas  of  satisfaction  between 
8.7  and  9.1  (on  a  scale  of  10). 


EXHIBIT  IV-58 


SOFTWARE  SUPPORT  PERFORMANCE 

STERLING 


SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.4 

Number  of  Major  Problems  Resolved  per  Month 

0.4 

Turnaround  Time  of  Major  Problems  (Hours) 

* 

Number  of  Minor  Problems  Reported  per  Month 

1.2 

Number  of  Minor  Problems  Resolved  per  Month 

1.2 

Turnaround  Time  of  Minor  Problems  (Hours) 

10.0 

*lnsufficient  Response 
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EXHIBIT  IV-59 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 

STERLING 


PERFORMANCE 

SATISFACTION  RATING* 

Freedom  from  Major 
Software  Problems 

Major  Problem 
Resolution 

Turnaround  Time  on 
Major  Problems 

PrppHnm  from  Minor 

1    1  Vm'  w  V.^  W  III    II  w  III    IVIII  1  w  1 

Software  Problems 

Minor  Problem 
Resolution 

Turnaround  Time  on 
Minor  Problems 

/  9.1 

^/////////////////// 

:  9.0 

8.7 

8.8 

8.9 

8.7 

Overall  Satisfaction 
with  Software  Support 

?  9.1 

1        23456789  10 


Low  High 
*Average  Standard  Error  of  the  Mean:  0.2 
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Exhibit  IV-60  (and  later  Exhibit  IV-62,  which  shows  the  user  reported 
value  ratings  versus  vendor  performance)  indicates  that  Sterling  users 
place  the  highest  value  on  their  software  engineer's  skill  level  (9.1  on 
scale  of  10),  documentation  (8.9),  and  phone  support  (8.7).  The  user 


EXHIBIT  IV-60 

SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

STERLING 




oUr  1  WAKE: 
SUPPORT 
CATEGORY 

LEVEL  OF  SUPPORT 

PERFORMANCE 

bAUbbUb 
(FALLS  BELOW) 
USER  VALUE 

VALUE 

PERFORMANCE 

On-Site  Support 

* 

* 

- 

Phone  Support 

8.7 

8.6 

(0.1) 

Accsss  to  Problems 
Data  Base 

7.8 

6.9 

(0.9) 

Documentation 

8.9 

7.9 

(1.0) 

Training 

7.5 

6.2 

(1.3) 

Consulting 

* 

* 

Software  Engineer 
Skill  Level 

9.1 

8.3 

(0.8) 

Scale:  1  =  Low,  10  =  High 

Average  Standard  Error  of  the  Mean:  0.5 

*lnsufficient  Response 
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reported  that  vendor  performance  did  not  meet  their  needs,  yet  these 
ratings  were  influenced  by  a  small  group  of  dissatisfied  respondents  that 
biased  the  performance  rating.  This  is  supported  by  Exhibit  IV-61,  which 
shows  that  Sterling  performance  satisfied  two-thirds  or  more  of  its  users 
in  each  of  these  valued  services.  In  fact,  the  only  weak  area  appears  to  be 
training,  where  only  47%  of  the  users  were  satisfied. 


EXHIBIT  IV-61 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 

STERLING 


SOFTWARE 
SUPPORT 
CATEGORY 

USERS  SATISFIED 
(Percent) 

On-Site  Support 

Phone  Support 

Access  TO 
Problems 
Data  Base 

Documentation 
Training 

Consulting 

Software  Engineer 
Skill  Level 

★ 

* 

0  20  40  60  80  100% 
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EXHIBIT  IV-62 

SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 

STERLING 


01        23456789  10 


VALUE 

A  =  Phone  Support  D  =  Training 

B  =  Access  to  Problems  Data  Base  E  =  Software  Engineer  Skill  Level 

C  =  Documentation 

Overall,  Sterling  users  are  quite  satisfied  with  the  support  that  they 
receive.  This  satisfaction  is  reflected  in  their  unwillingness  to  consider 
third-party  software  support  as  an  alternative  to  Sterling,  as  shown  in 
Exhibit  IV-63. 
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EXHIBIT  IV-63 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 
STERLING 


SUPPORT 
SERVICE 


Error  Fixes 

Phone  Support 

Problems 
Data  Base 

Consulting 
Training 


Willing  to  Consider 
Third-Party  Support 


7 


WILLINGNESS 
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UCCEL  Corporation  designs,  develops,  sells,  and  supports  both  applica- 
tions and  systems  software.  In  1987,  INPUT  surveyed  thirty  users  of 
UCCEL  software,  broken  down  by  the  following  products:  nineteen  users 
of  UCC-1  (a  disk/tape/file  management  system),  seven  users  of  UCC-7 
and  UCC-8  (two  communications  control  programs),  and  four  users  of 
UCC-11  (an  automated  rerun  and  tracking  system).  These  users  often 
used  these  programs  in  conjunction  with  other  UCCEL  systems  manage- 
ment software,  such  as  Space/Manager,  Tape/Manager,  and  System/ 
Manager. 

UCCEL  software  is  designed  to  run  on  IBM  (and  compatible)  main- 
frames. The  UCCEL  sample  ran  on  the  following  computer  systems: 
seventeen  IBM  30XXs,  twelve  IBM  43XXs,  and  one  Amdahl  58XX. 

The  software  analyzed  is  priced  as  follows:  UCC-1,  $26,000;  UCC-7, 
$65,000;  UCC-8,  $28,500;  and  UCC-11,  $21,000.  The  users  reported 
that  they  paid  between  12-15%  for  software  support,  averaging  14.4% 
per  year. 

Exhibit  TV-64  presents  that  software  support  picture  for  UCCEL  users. 
Most  support  is  delivered  by  telephone,  supplemented  by  access  to  the 
InfoPlus  problems  data  base.  Three-quarters  of  the  users  have  had  some 
level  of  on-site  support;  however,  few  contract  for  it,  and  UCCEl  reports 
that  at  least  75%  of  all  calls  can  be  handled  by  the  telephone.  UCCEL  is 
working  on  a  remote  support  system;  systems  software  users  can  access  a 
limited  level  of  remote  support  that  does  not  include  down-line  loading  of 
software  fixes. 

UCCEL  software  exhibits  a  high  degree  of  freedom  from  both  major  and 
minor  software  problems,  as  reported  in  Exhibit  IV-65.  UCCEL  users 
reported  so  few  major  problems  that  it  became  impossible  to  analyze 
major  problem  turnaround  time  with  any  statistical  accuracy.  Turnaround 
time  on  minor  problems  averaged  just  over  seventeen  hours,  well  under 
the  industry  average  of  26.5  hours. 
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EXHIBIT  IV-64 


SOFTWARE  SUPPORT  DELIVERY 
UCCEL 


SUPPORT 
DELIVERY 
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WHO  RECEIVE  SUPPORT  (Percent) 
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EXHIBIT  IV-65 


SOFTWARE  SUPPORT  PERFORMANCE 

UCCEL 


SUPPORT  COMPONENT 

Number  of  Major  Problems  Reported  per  Month 

0.2 

Number  of  Major  Problems  Resolved  per  Month 

0.2 

Turnaround  Time  of  Major  Problems  (Hours) 

* 

Number  of  Minor  Problems  Reported  per  Month 

1.0 

Number  of  Minor  Problems  Resolved  per  Month 

1.0 

Turnaround  Time  of  Minor  Problems  (Hours) 

17.1 

*lnsufficient  Response 


Needless  to  say,  UCCEL  user  satisfaction  with  each  of  these  areas  is 
relatively  high,  as  shown  in  Exhibit  TV -66.  Satisfaction  with  major 
problem  resolution  and  responsiveness  is  extremely  high,  ranging  be- 
tween 8.8  and  9.0  (on  a  scale  of  10);  however,  satisfaction  with  UCCEL 
performance  in  regards  to  minor  problems  (8.2-8.4)  tends  to  be  more  in 
line  with  the  entire  industry. 
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EXHIBIT  IV-66 


USER  SATISFACTION  WITH  SOFTWARE  SUPPORT  PERFORMANCE 

UCCEL 


PERFORMANCE 


Freedom  from  Major 
Software  Problems 

Major  Problem 
Resolution 

Turnaround  Time  on 
Major  Problems 

Freedom  from  Minor 
Software  Problems 
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Turnaround  Time  on 
Minor  Problems 


Overall  Satisfaction 
with  Software  Support 
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z 
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Exhibits  IV-67  through  IV-69  break  down  UCCEL  satisfaction  with 
software  support  in  finer  detail.  Exhibit  rV-67  indicates  that  UCCEL 
users  place  the  greatest  value  on  software  documentation  (rated  a  9. 1  on  a 
scale  of  10),  followed  by  software  engineer  skill  level  (8.5)  and  consult- 
ing (8.4). 


EXHIBIT  IV-67 


SOFTWARE  SUPPORT  VALUE/PERFORMANCE  LEVELS 

UCCEL 


SOFTWARE 
SUPPORT 
CATEGORY 

LEVEL  OF  SUPPORT 

PERFORMANCE 

EXCEEDS 
(FALLS  BELOW) 
USER  VALUE 

VALUE 

PERFORMANCE 

wll  Ollc  OUp|JUil 

5.2 

7.0 

18 

Phone  Support 

/.O 

O.O 

0.7 

Access  to  Problems 
Data  Base 

7.6 

7.5 

(0.1) 

Documentation 

9.1 

6.5 

(2.6) 

Training 

7.3 

6.7 

(0.6) 

Consulting 

8.4 

7.3 

(1.1) 

Software  Engineer 
Skill  Level 

8.5 

8.5 

0 

Scale:  1  =  Low,  10  =  High 

Average  Standard  Error  of  the  Mean:  0.5 
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UCCEL  users  reported  satisfaction  witii  tiieir  software  engineer  skill 
level:  74%  of  the  sample  received  at  least  the  level  of  support  in  this  as 
they  required,  as  shown  in  Exhibit  rV-68;  however,  UCCEL  users  are  less 
satisfied  with  the  consulting  services  and,  most  particularly,  documenta- 
tion that  they  received.  Only  44%  of  the  users  were  satisfied  with  con- 
sulting, which  they  often  felt  was  too  general  in  nature.  Of  greater  con- 
cern was  documentation,  which  satisfied  only  29%  of  the  users.  Three 
users  mentioned  documentation  as  the  most  important  improvement 
desired. 


EXHIBIT  lV-68 


SOFTWARE  SUPPORT  SATISFACTION  LEVELS 

UCCEL 
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(Percent) 
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Exhibit  rV-69  graphically  illustrates  the  discrepancy  between  users' 
requirements  for  dcx^umentation  and  consulting  services  and  UCCEL 
performance  and  how  this  discrepancy  has  led  to  user  dissatisfaction  with 
these  important  service  areas. 

EXHIBIT  IV-69 


SOFTWARE  SUPPORT  VENDOR 
PERFORMANCE  VERSUS  USER  VALUE 

UCCEL 

10 
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B  q/ 
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0        12  3 
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VALUE 

A 

=  On-Site  Support 

E  =  Training 

B 

=  Phone  Support 

F  =  Consulting 

C 

=  Access  to  Problems  Data  Base  G  =  Software  Engineer  Skill  Level 

D 

=  Documentation 
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Still,  UCCEL  users  are  very  happy  with  the  direct  support  in  response  to 
software  problems  that  they  receive  from  UCCEL.  This  satisfaction 
tends  to  limit  their  attraction  to  alternative  support  vendors.  Exhibit  IV- 
70  demonstrates  that  only  one  respondent  (3%  of  the  sample)  was  willing 
to  consider  a  third-party  software  support  vendor. 


EXHIBIT  IV-70 


WILLINGNESS  TO  CONSIDER 
THIRD-PARTY  SOFTWARE  SUPPORT 
UCCEL 


Willing  to  Consider 
Third-Party  Software  Support 
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1987  Software  Maintenance  User  Questionnaire 


Background 

Software  Vendor  

Software  Product  

And  you  receive  your  software  support  from  this  vendor? 
(If  no,  tliank  them  and  discountinue  the  interview.) 

Software  runs  on  (hardware  manufacturer/model)  

What  percent  of  software  purchase/license  cost  are  you  paying  for  support? 

 % 

Current  Software  Support 

Which  of  the  following  software  support  services  do  you  receive  on  a 
contract  basis  and/or  on  an  ad-hoc  basis?  (check  ail  that  apply) 

CONTRACT  AD-HOC 

a.  On-site     

b.  Telephone  (hotline)  support     

c.  Remote  support     

d.  Training     

e.  Consulting     

f.  Access  to  problems  data  base     
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Software  Support  Performance 

QU:7 

For  the  following  questions,  a  "major  software  problem"  can  be  defined  as 
one  in  which  processing  cannot  be  continued,  while  a  "minor  software  prob- 
lem" allows  processing  to  be  performed  with  minor  degradation.  All  ques- 
tions refer  to  the  last  12  months. 

a. 

Averaae  number  of  maior  problems  reported  (per  month) 

b. 

Number  of  maior  problems  resolved  (per  month) 

0. 

Turnaround  time  of  major  problem  resolution  (hours) 

d. 

Averaae  number  of  minor  problems  reported  (per  month) 

e. 

Number  of  minor  problems  resolved  (per  month) 

f. 

Turnaround  time  of  minor  problem  resolution  (hours) 

QU:8 

On 

a  scale  of  1  - 10  (10  being  highest),  how  satisfied  are  you  with: 

SATISFACTION 

a. 

Freedom  from  major  software  problems 

b. 

Major  problem  resolution 

c. 

Turnaround  time  on  major  problem  resolution 

d. 

Freedom  from  minor  software  problems 

e. 

Minor  problem  resolution 

f. 

Turnaround  time  on  minor  problem  resolution 

g- 

Overall  satisfaction  with  software  support 
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QU:9a 

Please  rate  your  requirement,  on  a  scale  of  1  - 10  (10  being  highest),  for  the 
following  software  support  services. 

QU:9b 

Please  rate  your  level  of  satisfaction,  on  a  scale  of  1  - 10,  for  the  service  that 
you  have  received. 

rv  n,     u  1 1\  ti  L/       i\  111       1  V  rL  JLf 

1 . 

vyii  out;  oupijuri 

2. 

Telephone  (hotline)  support 

3. 

Access  to  problems  data  base 

4. 

Documentation 

5. 

Training 

6. 

Consultinq 

7. 

Software  engineer  skill  level 

QU:10 

If  you  are  not  already  receiving  your  software  support  by  the  following  meth- 
ods, what  would  you  consider  to  be  a  reasonable  premium  to  pay,  as  a  per- 
centage of  your  current  charge,  to  receive  support  by  these  methods? 

1. 

PREMIUM 

On-site  support 

2. 

Telephone  (hotline)  support 

3. 

Remote  support 
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QU:  11a     Would  you  be  willing  to  consider  a  third-party  source  of  software  support? 

QU:  lib     (If  yes)  Please  rate,  on  a  scale  of  1  - 1 0  (1 0  being  high),  your  willingness  to 
use  the  third-party  support  vendor  for  the  following  services: 

WILLINGNESS 

1 .  Error  fixes   

2.  Telephone  (hotline)  support   

3.  Problems  data  base   

4.  Consulting   

5.  Training   

QU:  12      What  single  improvement  would  you  like  to  see  your  current  software 
support  vendor  make? 


Thank  You! 
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Appendix:  Definitions 


•  Applications  Software  -  Software  that  performs  processing  to  service 
user's  functions. 

•  Artificial  Intelligence  -  The  academic  discipline  involving  the  study  of 
the  processes  by  which  humans  perceive  and  assimilate  data  (and  use 
reasoning  to  process  this  data)  for  the  purpose  of  duplicating  these 
processes  within  computer  systems.  Also,  this  term  refers  to  the  com- 
puter systems  that  accomplish  these  duplicated  processes. 

•  BOC  -  Bell  Operating  Company. 

•  Consulting  -  Includes  analysis  of  user  requirements  and  the  develop- 
ment of  a  specific  action  plan  to  meet  user  service  and  support  needs. 

•  Dispatching  -  The  process  of  allocating  service  resources  to  solve  a 
support-related  problem. 

•  Divestiture  -  The  action,  stemming  from  antitrust  lawsuits  by  the 
Department  of  Justice,  which  led  to  the  breakup  of  AT&T  and  its 
previously  owned  local  operating  companies. 

•  Documentation  -  All  manuals,  newsletters,  and  texts  designed  to  serve 
as  reference  material  for  the  ongoing  operation  or  repair  of  hardware  or 
software. 

•  End  User  -  May  buy  a  system  from  the  hardware  supplier(s)  and  do 
own  programming,  interfacing,  and  installation.  Alternately,  may  buy  a 
turnkey  system  from  a  systems  house  or  hardware  integrator. 

•  Expert  Systems  Applications  -  Applications  for  expert  systems:  a  com- 
puter system  based  on  a  data  base  created  by  human  authorities  on  a 
particular  subject.  The  computer  system  supporting  this  data  base 
contains  software  that  permits  inferences  based  on  inquiries  against  the 
information  contained  in  the  data  base.  "Expert  systems"  is  often  used 
synonymously  with  "knowledge-based  systems,"  although  this  latter 
term  is  considered  broader  and  includes  expert  systems  within  its 
scope. 
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•  Engineering  Change  Notice  (ECN)  -  Product  changes  to  improve  the 
product  after  it  has  been  released  to  production. 

•  Engineering  Change  Order  (ECO)  -  The  followup  to  ECNs  which 
include  parts  and  a  bill  of  material  to  effect  the  change  in  hardware. 

•  Escalation  -  The  process  of  increasing  the  level  of  support  when  and  if 
the  field  engineer  cannot  correct  a  hardware  or  software  problem  within 
a  prescribed  amount  of  time,  usually  two  to  four  hours  for  hardware. 

•  Fiber  Optics  -  A  transmission  medium  which  uses  lightwaves. 

•  Field  Engineer  (FE)  -  For  the  purpose  of  this  study,  field  engineer, 
customer  engineer,  serviceperson,  and  maintenance  person  were  used 
interchangeably  and  refer  to  the  individual  who  responds  to  a  user's 
service  call  to  repair  a  device  or  system. 

•  Field  Service  Management  System  (FSMS)  -  A  specialized  application 
program  that  automates  some,  if  not  all,  of  the  following  activities  of  a 
field  service  organization:  call  handling,  dispatching,  parts  inventory 
and  tracking,  billing,  efficiency  reporting,  and  other  functions.  Ideally, 
the  system  accesses  one  data  base  from  which  each  function  can  use 
and  modify  data. 

•  Hardware  Integrator  -  Develops  system  interface  electronics  and 
controllers  for  the  CPU,  sensors,  peripherals,  and  all  other  ancillary 
hardware  components.  May  also  develop  control  system  software  in 
addition  to  installing  the  entire  system  at  the  end-user  site. 

•  ISDN  -  Integrated  Services  Digital  Network.  A  proposed  standard  for 
digital  networks  providing  transport  of  voice,  data,  and  image  using  a 
standard  interface  and  twisted  pair  wiring. 

•  LADT  -  Local  Area  Data  Transport.  Data  communications  provided 
by  the  BOCs  within  local  access  transport  areas  (LATA). 

•  Large  System  -  Refers  to  traditional  mainframes  including  at  the  low 
end  IBM  4300-like  machines  and  at  the  high  end  IBM  308X-like 
machines.  Large  systems  have  a  maximum  word  length  of  32  bits  and 
a  standard  configuration  price  of  $350,000  and  higher. 

•  Major  Software  Problem  -  Problems  ("bugs")  that  prevent  further 
processing. 

•  Mean  Time  Between  Failures  (MTBF)  -  The  elapsed  time  between 
hardware  failures  on  a  device  or  a  system. 
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•  Mean  Time  to  Repair  -  The  elapsed  time  from  the  arrival  of  the  field 
engineer  on  the  user's  site  until  the  device  is  repaired  and  returned  to 
the  user  for  his  utilization. 

•  Mean  Time  to  Respond  -  The  elapsed  time  between  the  user  placement 
of  a  service  call  and  the  arrival  at  the  user's  location  of  a  field  engineer. 

•  Microcomputer  -  A  microprocessor-based  single-  or  multi-user  com- 
puter system  typically  priced  less  than  $15,000.  A  typical  configuration 
includes  an  8-  or  16-bit  CPU,  monitor,  keyboard,  two  floppy  disk 
drives,  and  all  required  cards  and  cables. 

•  Minicomputer  -  See  Small  System. 

•  Minor  Software  Problem:  A  software  problem  ("bug")  that  allows 
further  processing  with  some  level  of  degradation. 

•  Operating  System  Software  (System  Software)  -  Software  that  enables 
the  computer  system  to  perform  basic  functions.  Systems  software,  for 
the  puiposes  of  this  report,  does  not  include  utilities  or  program  devel- 
opment tools. 

•  PBX  -  Private  Branch  Exchange.  A  customer  premises  telephone 
switch. 

•  Peripherals  -  Includes  all  input,  output,  and  storage  devices,  other  than 
main  memory,  which  are  locally  connected  to  the  main  processor  and 
are  not  generally  included  in  other  categories,  such  as  terminals. 

•  Planning  -  Includes  the  development  of  procedures,  distribution,  or- 
ganization, and  configuration  of  support  services.  For  example,  capac- 
ity planning,  "installation"  planning. 

•  Plug-Compatible  Mainframe  (PCM)  -  Mainframe  computers  that  are 
compatible  with  and  can  execute  programs  on  an  equivalent  IBM 
mainframe.  The  two  major  PCM  vendors  at  this  time  are  Amdahl  and 
National  Advanced  Systems. 

•  Professional  Services  -  A  category  of  services  including  system  design, 
custom  programming,  consulting,  education,  and  facilities  management. 

•  RBOC  -  Regional  Bell  Operation  Company.  One  of  seven  holding 
companies  coordinating  the  activities  of  the  BOCs. 

•  Remote  Diagnostics  -  Gaining  access  to  a  computer  from  a  point 
physically  distant  from  the  computer  in  order  to  perform  problem 
determination  activities. 
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•  Remote  Support  Implementation  -  An  extension  of  remote  diagnostics 
where  some  level  of  support  delivery  is  performed  from  a  point  physi- 
cally distant  from  the  computer.  Cuirently,  this  capability  is  more 
common  to  software  support  where  problems  can  be  solved  or  circum- 
vented through  downline  loading  of  new  code  (fixes). 

•  Reseller  -  A  marketing  organization  which  buys  long-distance  capacity 
for  others  at  wholesale  rates,  selling  services  at  retail  but  discounted 
prices  and  profiting  on  the  difference. 

•  Small  Business  Computer  -  For  the  purpose  of  this  study,  a  system 
which  is  built  around  a  Central  Processing  Unit  (CPU),  has  the  ability 
to  utilize  at  least  20M  bytes  of  disk  capacity,  provides  multiple  CRT 
workstations,  and  offers  business-oriented  systems  software  support. 

•  Small  System  -  Refers  to  traditional  minicomputer  and  superminicom- 
puter systems  ranging  from  a  small  multi-user,  16-bit  system  at  the  low 
end  to  sophisticated  32-bit  machine  at  the  high  end. 

•  Software-Defined  Network  -  A  private  network  which  uses  public 
network  facilities  and  which  is  configurable  on  an  as-needed  basis  by 
the  user  (see  Virtual  Private  Network). 

•  Software  Engineer  (SE)  -  The  individual  who  responds  (either  on-site 
or  via  remote  support)  to  a  user's  service  call  to  repair  or  patch  operat- 
ing systems  and/or  applications  software. 

•  Software  Products  -  Systems  and  applications  packages  which  are  sold 
to  computer  users  by  equipment  manufacturers,  independent  vendors, 
and  others.  Also  included  are  fees  for  work  performed  by  the  vendor  to 
implement  a  package  at  the  user's  site. 

•  Superminicomputer  -  See  Small  System. 

•  Systems  Integration  -  The  action  of  a  single  service  vendor's  design, 
development,  and  implementation  of  a  system  or  subsystem  including 
integration  of  hardware,  software,  and  communications  facilities  for  a 
customer. 

•  System  Interruption  -  Any  system  downtime  requiring  an  Initial  Pro- 
gram Load  (IPL). 

•  Systems  House  -  Integrates  hardware  and  software  into  a  total  turnkey 
system  to  satisfy  the  data  processing  requirements  of  the  end  user.  May 
also  develop  systems  software  products  for  license  to  end  users. 
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•  T-I  -  Refers  to  a  standard  1.544  megabit  per  second  digital  channel 
used  between  telephone  company  central  offices  and  now  used  for 
microwave,  satellite,  fiber  optics,  or  other  bypass  applications. 

•  Third-Party  Maintenance  (TPM)  -  Any  service  provider  other  than  the 
original  equipment  vendor. 

•  Training  -  All  audio,  visual,  and  computer-based  documentation,  mate- 
rials, and  live  instruction  designed  to  educate  users  and  support  person- 
nel in  the  ongoing  operation  or  repair  of  hardware  and  software. 

•  Turnkey  System  -  Composed  of  hardware  and  software  integrated  into 
a  total  system  designed  to  completely  fulfill  the  processing  require- 
ments of  a  single  application. 

•  VSAT  -  Very  Small  Aperture  Terminal.  A  small  satellite  dish  system, 
usually  using  Ku-band  frequencies. 

•  Virtual  Private  Network  -  A  portion  of  public  network  dedicated  to  a 
single  user. 
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